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^ | Abstract. Even if a probability distribution is properly normalizable, its associated 

■ Shannon (or von Neumann) entropy can easily be infinite. We carefully analyze 

I , conditions under which this phenomenon can occur. Roughly speaking, this happens 

when arbitrarily small amounts of probability are dispersed into an infinite number 
^ . of states; we shall quantify this observation and make it precise. We develop several 

particularly simple, elementary, and useful bounds, and also provide some asymptotic 
estimates, leading to necessary and sufficient conditions for the occurrence of infinite 
Shannon entropy. We go to some effort to keep technical computations as simple and 
' conceptually clear as possible. In particular, we shall see that large entropies cannot 

G ■ be localized in state space; large entropies can only be supported on an exponentially 

large number of states. We are for the time being interested in single-channel Shannon 
entropy in the information theoretic sense, not entropy in a stochastic field theory or 
. QFT defined over some configuration space, on the grounds that this simple problem 

' is a necessary precursor to understanding infinite entropy in a field theoretic context. 

o ' 
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1. Introduction 

The classical Shannon entropy, (and closely related quantum von Neumann entropy), is 
a general purpose theoretical tool with a vast number of applications ranging from 
engineering to demographics to quantum information theory and other branches of 
theoretical physics [U El El IU El E]- Stripped to its essentials, one considers a set of 
normalized probabilities ^2 n p n — 1> and analyzes the properties of the quantity: 

S = -^p n lnp„. (1) 

n 

Two major cases are of interest, when the index set {n} characterizing the various 
"states" of the system is finite, and when it is countably infinite. A third case, when the 
index set {n} is uncountably infinite, requires an integral formulation of the entropy, 
and we shall not presently have anything specific to say about this uncountable case. 
One way of justifying such a restriction is via an appeal to quantum mechanics where, 
in terms of a normalized density matrix tr[p] = 1, the von Neumann entropy is: 

S= -tr[plnp]. (2) 

If, (as is usual), quantum physics is formulated on a separable Hilbert space, then the 
density matrix can be diagonalized over a countable basis, and the von Neumann entropy 
reduces to the Shannon entropy over a countable (or possibly even finite) set of states. 
For this reason we shall restrict attention to the finite or countably infinite cases. 

If the total number of states is finite, N = Yl n 1 < oo, then an elementary 
computation leads to the upper bound 

S < In N. (3) 

More subtly, even if the total number of states is infinite, N = J2 n 1 = oo, then as long 
the total number of states of non-zero probability is finite, N' = J2n Pn >o 1 < oo, an 
equally elementary computation leads to the upper bound 

S<\nN'. (4) 

These simple observations demonstrate that to obtain infinite Shannon entropy, an 
infinite number of states must have non-zero probability, in particular: 

N' > exp S. (5) 

But then, since the sum of the probabilities is unity, an infinite number of states must 
have non-zero but arbitrarily small probability. We shall now seek to quantify these 
observations in a straightforward and transparent manner. Some earlier rather technical 
work along these lines can be found in [7J . Our own interest in these issues was prompted 
by the more general issues raised in P El [101 IU H21 UHl HU EEl [161 IE]- I n the long 
run we are interested in infinite entropies that can arise in stochastic field theories and 
QFTs, but the single channel information theoretic context of the current article already 
provides some interesting subtleties. 
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2. Some examples of infinite Shannon entropy 

To show that situations of infinite Shannon entropy can indeed occur perhaps the 
simplest example is to consider the sum: 

Tj(u) = y — \TT — 5 (converges for u > 0, diverges for u < 0). (6) 

Variants of this series are discussed for instance in Hardy [IB] . Hardy and Riesz [T9J, see 
esp p 5], and Shilov [20j see es P §6.15.c on p 192]. Here \x] is the "ceiling" function, 
the smallest integer > x. The perhaps somewhat unexpected lower limit of summation 
\e] is designed to ensure that In n > for all terms in the sum, so that one never has to 
raise a negative number to a real power. The corresponding probabilities (defined only 
for n > \e\) are 

Pn = E(u) n (lnn)^"' ^ 
These are well defined and properly normalized for u > 0. But then 

S = E w S n , 1+M ln{S( M )n(lnn) 1+ »} (8) 
' h(u) n Inn 1+M 

n \ / \ / 

1 ^-^ 1 1 + u sr-^ In In n 

11 ^ + £(V) ^ n (lnn)« + Y^uj ^ n (lnn) 1+ « ^ 

1 . i 1 1 dS(M)/d« , . 

= hEu + — ) — r — • 10 

V ; E(w) ^ n (lnn)« E(u) V ; 

The first and third terms converge for u > 0, but the second term converges only for 
u > 1. So this probability distribution is convergent but has infinite Shannon entropy 
over the entire range u G (0,1]. The particular case n — 1, where p n oc n ^ n )z aa,s 
previously been considered in a number of specific situations [T0| 121] . Apart from the 
entire range u G (0, 1] above, there are many more examples along similar lines. For 
instance one could consider the sums 

= - i nn (i n i nn u + « ; 



n=\e e ~\ 



x 

z — ' n inn In Inn Inmmn i+u 

r f=e -t V / 

n=|e e | 

both of which converge for u > and diverge for u < 0, and the obvious infinite chain of 
generalizations thereof. Thereby (following the analysis above) one can easily construct 
an infinite chain of probability distributions that are convergent (and so are properly 
normalized) for u > but whose Shannon entropy converges only for u > 1. These 
probability distributions are all convergent but have infinite Shannon entropy over the 
entire range u G (0,1]. Even more baroque examples are possible, (but perhaps not 
desirable) . 
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3. Probability gap 

As a first step towards analyzing and quantifying the conditions under which infinite 
Shannon entropy can occur, let us define a notion of "probability gap" when there is a 
minimum non-zero probability. (The idea is to mimic the notion of "mass gap".) More 
precisely, let 

= inf{p n : p n > 0}. (13) 

If > then, (since 1 = J2 n Pn — p*), we have N' < 1/p* < oo. We see that only a 
finite number of the p n can then be non-zero, and in fact the infimum can be replaced 
by a minimum: 

p* = min{p n : p n > 0}. (14) 

So for all practical purposes the presence of a probability gap means the state space is 
effectively finite. Conversely if only a finite number of probabilities are non-zero then 
there is a probability gap. In particular it is elementary that 

S = ~y^p n lny n < -^p n lnp* = - hip* < oo, (15) 

n n 

though a slightly stronger result is also available 

S < IniV' < -hip* < oo. (16) 

So we see very explicitly that for infinite Shannon entropy one cannot have a probability 
gap. 

4. Elementary bound leading to the Gibbs inequality 

Let us now try to be more quantitative. Based on the utterly elementary fact that for 
positive x we have [xlnx]" = 1/x > 0, it is immediate that for positive numbers 

(x ~\~ y \ 
'a + b) ' ^ 

with equality only when x/a = y/b. See [221 P 97 §H7]- 
Proof: Since the second derivative is positive 

a _, b ax + by f ax + by 
x In x H ymy> — In 



a + b a+b ~ a+b \ a+b 

with equality only when x = y. Therefore 

~i~ 7~i ~ / ~ ~\ f ax + by\ 

ax m x + by In y > (ax + by) In — J . (19) 

\ a + b J 

Now rename ax — > x and by — )■ y to obtain the desired result. □ 
(It is worth explicitly verifying this since the justification is so elementary, and the payoff 
will be immediate.) Now iterate this result: 

xln(x/a) +y\n(y/b) +z\n(z/c) > (x + y + z) In ■ ( 20 ) 
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More generally, for positive x n and a n : 

£ *„ ln(x n /a n ) > ( £ ) In (^^) . (21) 

V n / \^n a nj 

When J2 n Pn — J2 n In = 1 the above gives an elementary proof that 



J^PnHPn/qn) > [Y^Pn J In (^f) = 0. 



That is 



so 



(22) 



£p n lnp n > £p n lng n , (23) 



5<-£p„lng„. (24) 

n 

This result is of course extremely well known, typically being referred to as the Gibbs 
inequality, (or the positivity theorem for relative entropy), with proofs most typically 
involving a less than elementary appeal to Jensen's inequality. We shall now apply this 
result to the matter at hand. 



5. Partial counts, partial probabilities, partial Shannon entropies 

Let us now consider the effect of summing only over some restricted subset X of the 
total state space {n}. Define 

N X = J2^ Px = J2p n ] S x = -Y J Pn^Pn- (25) 

neX neX neX 

In particular, using the inequality demonstrated above, we have 

J> ra lnp„ > ( 5> n ) In ( ^ nexPn ) = P x \n{P x /N x ). (26) 

neX \neX J ^ l^ndX 1 J 

Therefore 

S x <P x [\nN x -\nP x ]. (27) 

Though this looks very similar to the entropy bound derived for the total entropy over 
a finite state space, there are significant differences — the current bound now tells you 
something deeper about the extent to which entropy can be localized in the state space. 
Indeed we can recast the bound as: 

N x > P x exp(S x /P x ). (28) 

That is, packing a finite amount of entropy S x into a region containing total probability 
P x requires an exponentially large number of states N x . 

Another way of interpreting this bound is to define the average probability per 
state, and average entropy per state, over the set X by: 

- Pv - S V . 
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Then 

S x <-P x \nP x . (30) 

A slightly weaker but perhaps more intuitive bound is obtained if we vary the RHS of 
equation ([27]) with respect to P x while holding N x fixed (assume N x > \e] = 3). Then 
a RHS 



\n(N x /P x )-l>0. (31) 
or x 

So the maximum of the RHS occurs for P x = 1, and we see (for N x > 3) 

S x <\nN x . (32) 

Similarly if we vary the RHS of equation (128]) with respect to P x while holding S x fixed 
(assume S x > 1). Then 

= exp(S x /P x ) {1 - S x /P x } < 0. (33) 

af x 

So the minimum of the RHS occurs for P x = 1, and we see (for S x > 1) 

N x >expS x . (34) 

The message to take from the logarithmic and exponential bounds is again that large 
Shannon entropies cannot be tightly localized in state space, large Shannon entropies 
must invariably come from exponentially large (N x > exp S x ) regions of state space. 

6. Asymptotic estimates 

Let us now consider the effect of adding some extra order-based structure, by summing 
only over the high-probability sector of the total state space {n}. Define the quantities: 

N( P )= !; p (p)= E p»; s (p) = - E ^ ln ^«- ( 35 ) 

n:p n >P n:p n >p n:p n >p 

These are "probability cutoff" sums where the low probability events are excluded. Note 
\imN(p)^N; limP(p)->l; Urn S(p) -> S; (36) 

p— >0 p— >0 p— >0 

where N and S may be infinite. It may sometimes be useful to define N' = lim p _ 5>0 + A(p) 
(which may again be infinite) in order to explicitly exclude the zero modes. Now 

S(p) < p n [— In p] = —P(p)\np < —In p. (37) 

n:p n >p 

That is 

S(p)<-\np, (38) 

so in some sense the total Shannon entropy can never be worse than "logarithmically 
divergent" in the probability cutoff. Similarly 

P(v) 1 

1 > P(p) > pN(p); that is N(p) < — ^ < -. (39) 

p p 
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We also have 

S(p) < P(p)[\nN(p) - lnP(p)]. (40) 

Combining these results we regain 

S(p) < -P(p) hip < - In p. (41) 

We again see that to get infinite Shannon entropy one needs an infinitely large number 
of arbitrarily low probability events. 



7. Entropy bounds from the Gibbs inequality 

Let us now obtain several explicit bounds directly from the Gibbs inequality. Consider 
the quantities q n = n~ z /((z) where ((z) is the Riemann zeta function. Then we have 
Yln°=i Qn — 1 f° r z > 1- The Gibbs inequality becomes 

oo oo 

S <-^p n \nq n = \n((z) + z^p n \nn. (42) 

n=l n=l 

Thus a sufficient condition for the Shannon entropy to be finite is 

oo 

(Inn) = ^^p n Inn < oo. (43) 

71=1 

A number of quite similar results can easily be obtained: 

• Consider for instance the quantity E(e) = X^i ex P(~~ n<i )- This sum is convergent 
when e > 0. Then set q n = exp(— n e )/£(e), and note Y^=iQn — 1 provided e > 0. 
Then the Gibbs inequality becomes 

oo oo 

S<-^P„mg„ = lnE(e) + ^p n n e . (44) 

n=l n=l 

Thus a sufficient condition for the Shannon entropy to be finite is that there exist 
some e > such that 

oo 

( n ') = ^Pnrf < OO. (45) 

71=1 

This is of course part of a general phenomenon. 

• Let E n be a collection of numbers such that Z((3) = X^i ex P( — PE n ) converges for 
some at least one value of (3. Now define q n = exp(—j3E n )/Z(j3), then Yln^i Qn = ^ 
provided (5 is such that the sum Z{0) converges. Then the Gibbs inequality becomes 

oo oo 

S < - Y,Pn lng„ = InZ(P) + (3Y,PnE n . (46) 

n=l n=l 

Thus a sufficient condition for the Shannon entropy to be finite is that there exist 
some set of numbers E n , and some (3, such that the corresponding Z((3) converges 
and such that 

oo 

(E n ) = Y,PnE n <oo. (47) 

71=1 
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On the other hand, deriving a necessary condition requires rather different tools. Let 
us first re-order (if necessary) the p n so they are in non- increasing order (p n+ i < p n ). 
Then 

m m 

1 > ^Pn > ^Pra = ™P m - (48) 
n=l n=l 

That is, with this choice of ordering, we are guaranteed p n < 1/n. But then 

oo oo 

S = - ^2p n \np n > ^p n lnn. (49) 

n=l n=l 

Thus a necessary condition for the Shannon entropy to be finite, when the probabilities 
are sorted into non-increasing order, is that 

oo 

(Inn) = ^^p n lnn < oo. (50) 

n=l 

We can eliminate the need for explicit re-ordering as follows: Using the previously 
defined object N(p) = J2n- P „> P 1 we can define the quantities 

p n = m&x{p : N(p) > n}. (51) 

Then p n is automatically a rearrangement of the p n in non- increasing order, and a 
necessary condition for the Shannon entropy to be finite is that 

oo 

(lnn)~ = ^^Pn Inn < oo. (52) 

n=l 

The mathematical tools used so far have been extremely basic inequalities and series; the 
analysis has been minimal. We shall now use some slightly more sophisticated analysis 
in the form of Dirichlet series. 

8. Dirichlet series 

Define the generalized zeta function, a particular type of Dirichlet series [19], by 

oo 

Cs(z) = J2(PnY- (53) 

71=1 

One could think of the S as standing either for Shannon or for entropy. A minor 
improvement is to explicitly exclude any states of zero probability and take 

n:p n >0 

By construction Cs^l) = Y^=iVn = 1, so this particular Dirichlet series certainly 
converges (absolutely) for z > 1. The interesting question is whether it converges 
for any z less than 1. Note that 

dlnC 5 (^ 



S - d* 



(55) 

2 = 1 
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If we now view zasa complex number then, (in contrast to the usual situation for Taylor 
series where there is a radius of convergence), for Dirichlet series there is an abscissa of 
convergence a such that the series converges over the complex half-plane defined by [19] 

> a, (56) 

and diverges over the complex half plane defined by 

< o. (57) 

The line $l(z) = o has to be treated delicately, in a manner somewhat analogous to 
the fact that that Taylor series behavior on the radius of convergence has to be treated 
delicately [T9l see esp p 10]. The fact that the series is clearly convergent for real z > 1 
guarantees that a < 1, the abscissa of convergence is bounded above by unity. The 
relevance of this observation lies in the following fact: 

A sufficient condition for the entropy to be finite is that a < 1. (58) 

For a finite state space this is automatic. If we take the definition where zero probability 
states are excluded then the abscissa of convergence is o = — oo. (Even if we somewhat 
foolishly keep the zero probability states in the Dirichlet series, we still have a = 0.) For 
a countably infinite state space there is something to be proved. In particular, because 
all the coefficients in the generalized zeta function Cs( z ) are positive, the real point on 
the abscissa of convergence is known to be a singular point of the function (s{z)- See [19j 
see p 10]. The word "singular" is used in the sense of "not analytic", so that there is no 
convergent Taylor series around the point z = a. This happens if (for sufficiently large 
m) one of the derivatives diverges: 

oo 
n=l 

If this happens for m — 1 (the first derivative) then the entropy is infinite. However, 
this might not happen until m > 1, perhaps even much greater than 1. That is: 

Unfortunately a < 1 is not a necessary condition for finite entropy. (60) 

Example 1: As an explicit example of this phenomenon, recall that we had previously 
seen that the particular choice 



P„ 



£(«) = V —z rj—; (61) 



S(«) n (lnn) 1+n ^ i n(\nn) 1+u 



leads to both finite entropy and normalizable probability for u > 1. But the generalized 
zeta function corresponding to this particular p n is 

CsM = =(«)- E „,( h „)(^„ - < 62 > 

And for this particular zeta function it is very easy to see that the abscissa of convergence 
is a — 1. (See for instance related discussion in Hardy [IB] , Hardy and Riesz [T9J, or 
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Shilov [2U]; the key point is that for $t(z) 7^ 1 the n z term dominates and controls 
convergence/divergence of the series. For dt(z) = 1 one has to look carefully at the 
exponent of the Inn term.) Furthermore, for this particular p n we see 

cr (i) = ± - m m + ^ E ""Iff" 1 " ' < 63 > 

The dominant term in this last sum comes from the (lnn) m in the numerator, so 
convergence of Q (1) f° r the specific probability distribution presented in equation 
(1ST]) is equivalent to convergence of 
00 1 

V - (64) 

^ n(\\in)^ +u ~ m ) y ' 

But this series converges only for u > m. So even if the probabilities converge (u > 0), 
and even if in addition the entropy converges (u > 1), for any finite u there will always 
be a sufficiently high derivative (m > w) that fails to converge. This verifies by explicit 
example that a < 1 is not a necessary condition for finite entropy. □ 

Example 2: On the other hand, let us now consider the following situation: Let Zq > 1, 
and define quantities 

Pn = (65) 

Then by construction J2 n Pn = 1 is absolutely convergent. The generalized zeta function 
associated with p n is 

? , s \- ( (Pn) Z0 Y (s(zqz) 

But this implies (s( z ) is convergent for z z > 1, that is z > l/z . Therefore the abscissa 
of convergence for the p n satisfies a < 1/zq < 1, which implies that the p n are guaranteed 
to have finite Shannon entropy. Now 

InCs(s) = In Cs(zqz) - z\nCs(z ). (67) 
A brief computation yields 



§ = _ d(s(z) 



dz 



din (s(z) 

Zo 



dz 

z=l 



+ ]n( s {z ). (6? 



2=Z0 

This is certainly finite for any z > 1. So it is extremely easy to construct a very large 
class of probability distributions which have both finite Shannon entropy and and an 
abscissa of convergence strictly less than unity: a < 1. □ 
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9. Summary and Discussion 

We have considered situations of infinite entropy, (primarily Shannon entropy, though 
the modifications required for dealing with von Neumann entropy are straightforward), 
defined over a countably finite state space. The discussion concerns single- channel 
information theoretic entropy, and the additional subtleties encountered in stochastic 
field theories and QFT are deferred for now. We have developed a number of very simple 
bounds and asymptotic estimates to probe the onset of infinite Shannon entropy, with 
an emphasis on keeping technical computations as simple as possible. Key results are 
that to obtain infinite Shannon entropy an infinite number of states must have non-zero 
but arbitrarily small probability, that the Shannon entropy can never be too divergent, 
and that in a suitable technical sense infinite Shannon entropy is never worse than 
logarithmic in the cutoff. The message to take from this logarithmic bound is that large 
Shannon entropies cannot be tightly localized in state space, large Shannon entropies 
must invariably come from exponentially large regions of the state space. 
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